DOCUMENT RESUME 



ED 465 333 



HE 034 983 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Snyder, Vivian; Hackett, Rachelle Kisst; Stewart, Mark; 
Smith, Douglas 

Predicting Academic Performance and Retention of Private 
University Freshmen in Need of Developmental Education. 
2002-04-00 

22p.; Paper presented at the Annual Meeting of the American 
Educational Research Association (New Orleans, LA, April 
1-5, 2002) . 

Reports - Research (143) -- Speeches/Meeting Papers (150) 
MFOl/PCOl Plus Postage. 

♦Academic Achievement; College Entrance Examinations; 
♦College Freshmen; ♦Developmental Programs; Grade Point 
Average; High School Students; Higher Education; 
♦Prediction; Private Colleges ; ♦School Holding Power; 
Scores; Sex Differences 



ABSTRACT 



This study examined the academic success and retention of 
first-year college students at a small private university to determine which 
variables (among the following: gender, high school grade point average 
(GPA) , Scholastic Assessment Test (SAT) verbal and quantitative scores, and 
indicators of placement into developmental courses) were predictive of three 
outcomes: (1) academic achievement during the freshman year; (2) retention to 

the, second year; and (3) the combination of retention and university GPA 
greater than or equal to 2.5. Results for 666 students in 3 years confirm the 
findings of other studies, in which GPA at the end of the first year is 
associated with gender, high school GPA, and verbal and quantitative SAT 
scores; however, the type of skills development needed by the student--a 
course in reading, writing or mathematics- -was not predictive of academic 
achievement or retention. These findings are discussed in light of the 
developmental program offered at the institution. (Contains 1 figure, 4 
tables, and 22 references.) (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



QH 



Predicting Academic Performance and Retention of Private University 
Freshmen in Need of Developmental Education 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



Dr. Vivian Snyder 
Educational Resource Center 
University of the Pacific 
Stockton, CA 9521 1 

vsnvdei'@Liop.edu 

209/946-3218 

Dr. Rachelle Kisst Hackett 
Benerd School of Education 
University of the Pacific 
Stockton, CA 95211 

rhackeu@iiop.edu 

209/946-2678 
Dr. Mark Stewart 

International Programs and Services 
University of the Pacific 
Stockton, CA 9521 1 

mslewait@uop.edu 

209/946-2247 

Dr. Douglas Smith 
Department of Computer Science 
University of the Pacific 
Stockton, CA 9521 1 

dsmith@uop.edu 

209/946-3036 



educational resources information 
center (ERIC) 

IS document has been reproduced as 

oS.f 'T or^o?gan?zatfon 

originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



doiumplrlf stated in this 
document do not necessarily represent 
official OERI position or policy. 




Paper Presented at the Annual Meeting of the 
American Educational Research Association 
April, 2002 



2 



MST COPY AVAILABLE 



Abstract 



The study examined the academic success and retention of first-year college students at a 
small private university to determine which variables (among the following set — gender, high 
school GPA, SAT verbal and quantitative scores, and indicators of placement into developmental 
courses) were predictive of three outcomes: (1) academic achievement during the freshman year, 
(2) retention to the second year, and (3) the combination of retention and university GPA >2.5. 
Results confirm the findings of other studies, where GPA at the end of the first year is associated 
with gender, high school GPA, and verbal and quantitative SAT scores. However, the type of 
skills development needed by the student — a course in reading, writing, and/or mathematics — 
was not predictive of academic achievement nor retention. These findings are discussed in light 
of the developmental program offered at the institution. 
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PREDICTING THE ACADEMIC PERFORMANCE AND RETENTION OF UNIVERSITY 
FRESHMEN IN NEED OF DEVELOPMENTAL EDUCATION 



Introduction 



Purpose and Rationale 

This study looked at the academic achievement and retention of university freshmen in 
need of developmental education (i.e., those who test into one or more developmental courses). 
The university (a small — 4,000 students, private, liberal arts, California school) shares a goal 
with most postsecondary education institutions, i.e., that of improving retention, especially as 
indicated by the return of freshman students as sophomores. 

Predictors readily available to admissions officers at institutions have included the 
student’s gender, prior educational performance, and scholastic aptitude test scores. The 
predictability of high school grades has long been known (Astin, 1993; Astin, Tsui, & Avalos, 
1996; Gallicki & McEwen, 1989; Lewallen, 1993; Daly & Breegle, 1989; and York, Bollar, & 
Schoob, 1993). 

Once students are admitted to the university where the current study took place, they take 
basic skills proficiency tests in reading, writing, and quantitative (mathematics) skills during 
orientation prior to the first day of class. Based on testing, these developmental students are 
required to take one to three developmental education courses. Clifford Adelman, of the U. S. 
Department of Education, has examined data from the college transcripts of a sample of 1982 
high school graduates, with follow-up surveys through 1993. Adelman reports that, of this 
group, the students who took the greater number of developmental courses had lower rates of 
bachelor’ s-degree completion. He also found that students who entered college needing 
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developmental work in reading were more likely to fail than those who required work in 
mathematics or writing (Adelman, 1996, p. A56). 

In this study we investigated whether a reasonably accurate prediction model can be 
developed for use with the developmental student population (i.e., that population of freshman 
students placing into one or more of the developmental education courses) that is based only on 
information gathered prior to admission. We also tested whether knowing the type of 
developmental education needed (whether reading, writing, or math) improved the prediction. 
The aim was to develop a model for the classification of future students and to determine the 
accuracy of the model’s prediction using a cross-validation sample of similar university students 
who required developmental education courses. One use of this model would be to identify 
developmental students who were not predicted to be successful, even given the opportunity for 
academic support, and encourage their utilization of services. 

The purpose of the current study was to determine, for developmental students, which of 
a set of variables were predictive of three outcomes: (1) academic achievement during the 
freshman year, (2) retention to the second year, and (3) a combination of retention and GPA > 
2.5. Moreover, the aim was to develop a model for the classification of future students and to 
determine the accuracy of the model’s prediction using a cross-validation sample of similar 
university students who required developmental education courses. 

Method 

Sample 

The population consisted of freshmen admitted to a small, private university whose 
performance on basic skills tests taken during orientation (after students’ admission to the 
university) indicated a need for developmental education. Based on testing, these developmental 



students are required to take one to three developmental education courses. International 
students and students allowed to defer enrollment in what would otherwise be a required 
sequence of freshmen courses at our university (known as “Mentor” classes) were excluded from 
the sample. Deferrals to the Mentor classes are on the basis of extremely low reading and/or 
writing scores (typically so that students can enroll in ESL courses before their developmental 
reading and/or writing courses). Thus Mentor-deferred students are not enrolled in the same 
curriculum as other freshmen at the institution. 

The population for which records were accessible includes developmental students at the 
university (who were neither international nor Mentor-deferred students) who matriculated 
during the Fall semesters of 1995 (n= 260), 1996 (n= 162), and 1997 (n= 244). Thus, the sample 
available for use in the data analysis consisted of 666 students. 

The sample was predominately female (59.8%), and most of the students (80.3%) 
enrolled the following Fall Semester (i.e., cases we classified as “retained”). The proportion of 
students in the sample who were required to take developmental courses follow: Reading- 
26. 6%; Writing- 70.6%; and Math- 47.1%. Descriptive statistics for the sample’s high school 
grade point average, university grade point average, and SAT scores are shown in Table 1 . 
Developmental Course Placement 

All students who are admitted to the University are administered basic skills tests during 
the summer Freshman Orientation sessions, prior to their matriculation Fall semester. Placement 
into the developmental reading course is based on a total raw score of 91 or below on the Nelson 
Denny Reading Test, Form G (Brown, Fishco, & Hanna, 1993). Placement into the appropriate 
developmental math course based on scores achieved from taking one or more tests from The 
Descriptive Tests of Mathematics Skills of the College Board (Educational Testing Service, 
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1979). And placement into the appropriate writing course was based on a score achieved by a 
locally-developed, holistically-scored writing sample, read by two readers (or a third reader in 
the case of disparate scores). 

Variables 

The seven predictor variables included students’ gender (1= female, 0= male); high 
school grade point average (HSGPA), SAT college entrance exam scores (Verbal: VSAT; 
Quantitative: MSAT), and whether the student was required to take developmental reading, 
writing, or mathematics (1= developmental course required, and 0= no developmental course for 
READING, WRITING, MATH). Criterion variables included students’ university grade point 
average (U-GPA) after two semesters, retention status (RETAINED: 1 =enrolled the following 
Fall Semester, 0=not enrolled), and a combination (GPAOKRET) of these (l=retained with a 2.5 
U-GPA or better, 0=not retained and/or U-GPA under 2.5). 

Data Analysis Procedures 

The database was created by combining archival information available from the 
university’s Office of the Registrar and its developmental program, the Educational Resource 
Center. Once the records of the 666 cases in the sample were available, a sample of 
approximately 70% of the cases (N= 462) was randomly selected to use in the model building 
phase of the analysis. The remaining cases (N=204) were held out for use in the model 
validation phase. Listwise deletion of missing cases (i.e., where a case is removed if information 
on any one of the predictor or criterion variables is unavailable) was employed leaving 426 cases 
for the initial phase. 

Pearson product-moment correlations were calculated for each predictor variable- 
criterion variable pair (see Table 2) and tests of statistical significance were employed using an 



alpha level of .01 (1 -tail) to determine whether the two respective variables are related. As Glass 
and Hopkins (1996) explain, “The general formula for the Pearson r is applicable when either or 
both X and Y are ranks or dichotomous, but the resulting coefficients are denoted by special 
names. . . . When one variable is a dichotomy, the correlation is termed a point-biserial 
coefficient; if both variables are dichotomies, the correlation is a phi coefficient” (p. 140). These 
results are mathematically consistent with those obtained had /-tests for independent samples 
replaced testing of point-biserial coefficients and chi-square tests of association replaced testing 
of phi coefficients. See the appendix for a table of corresponding statistical tests. 

Logistic regression was utilized to develop and test the prediction model. The final 
logistic regression that best fit the data included just two predictors. Because 19 of the 36 cases 
originally removed through listwise deletion with the full 7-predictor model were missing values 
only for predictors that did not become part of the final model, the latter reduced model is based 
on the 445 cases with valid values for GENDER, HSGPA, and GPAOKRET (the criterion). In 
the validation sample, 183 cases (of 204) had complete information and were used to check for 
shrinkage in the classification accuracy. 

Results 

Individual Predictors of Freshman Academic Achievement 

Statistical significance testing of the correlations associated with the first outcome 
indicate that the university GPA’s at the end of the freshman year are associated with gender, 
high school GPA, verbal SAT scores, and quantitative SAT scores. Knowing which of the basic 
skills tests the developmental student failed was not predictive of freshman GPA. (See the first 
column of numerical entries in Table 2, which shows the correlation coefficients between 
individual predictors and the three outcomes.) 
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Individual Predictors of Freshman Retention 



Statistical significance testing of the correlations associated with the second outcome 
indicate that high school GPA is the only predictor in this study that is associated with whether 
or not students enroll at the university for a second year (i.e., one year later in the Fall Semester). 
Neither gender, SAT scores, nor basic skills indicators were predictive of retention. (See the 
second column of Table 2.) 

Individual Predictors of Freshman Retention with University GPA > 2.5 

Statistical significance testing of the correlations associated with the third outcome 
indicate that gender, high school GPA, and SAT quantitative scores are each associated with 
whether or not students enroll at the university for a second year (i.e., one year later in the Fall 
Semester) and have a university GPA of 2.5 or better. Neither verbal SAT scores, nor basic 
skills indicators were predictive of the combined outcome based on retention and academic 
performance. (See the third column of Table 2.) 

Logistic Regression Model for Predicting Freshman Retention with University GPA ^ 2.5 

A model for predicting the probability that an incoming freshman would be retained for a 
second year and have achieved a GPA ^2.5 his/her first year was developed using logistic 
regression. Students who met the criterion of being retained with a GPA ^ 2.5 were coded 1 
(referred to as “Group 1” below); students who either were not retained (by their own choice or 
by failing to meet university requirements) or who did not achieve a cumulative GPA during 
their freshman year of at least 2.5 were coded 0 (referred to as “Group 0” below). 

Coincidentally, when coupled with retention status, the outpoint of 2.5 we assigned for 
separating students by GPA below and above 2.5 resulted in equal numbers of cases in each of 
the dichotomous groups, which together constitute the criterion being predicted. This is 
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fortunate, because, as Wright (1995) points out, having proportions of cases in each group that 
are approximately equal typically optimizes classification accuracy. We used a non-sequential 
logistic regression where all 7 predictors were initially included in the model. Based on the 
Wald test, however, only gender and high school GPA were found to be statistically significant. 
The deviance (-2LL) was 547.379. Although the model with these two predictors was an 
improvement over the base model which utilizes only a constant term [Model chi-square(2 df, 
N= 445)= 69.502, p<.001], the Nagelkerke R Square value of .193 is modest. Sensitivity (the 
proportion of Group 1 that was correctly classified) was 70.6%. Specificity (the proportion of 
Group 0 that was correctly classified) was 66. 1%. The overall percentage of correctly classified 
cases (PAC) using gender and high school GPA as predictors was 68.3% which, although low, 
compares favorably to the base model’s rate of 50.3% when no predictors are utilized. 

The 2-predictor model is presented in Table 3. The model predicted that, when 
controlling for high school GPA, the odds of being retained with a GPA ^ 2.5 increase by a 
factor of 1 .84 when the student is female (as compared to male). After controlling for gender, 
for every positive one unit of change in high school GPA (for example, a “3.3” versus a “2.3” 
GPA), the odds of being in the target group increase by a factor of 6.65. The two logistic curves 
(one for each gender) based on the model are shown in Figure 1 where it can be seen that the 
high school GPA cutoff that places a student into the target group is lower for females (3.1 1) 
than it is for males (3.43). It should be noted that, although significantly correlated with the 
criterion (See Table 2.), the quantitative SAT score is not a significant predictor after controlling 
for gender and high school GPA. 




10 



9 



Classification Accuracy of the Model Based on a Validation Sample 



Using the model coefficients based on the analysis sample which are reported in Table 3, 
predicted probabilities for membership in the target group were calculated for each case in the 
validation sample. Those cases with probabilities of .5 or better were predicted to be retained 
with GPA ^ 2.5; those remaining cases were classified into Group 2. (See Table 4.) Sensitivity 
was 67.0% (as compared to 70.6% in the analysis sample). Specificity was 57.6% (as compared 
to 66.1% in the analysis sample). The overall percentage of correctly classified cases (PAC) 
using gender and high school GPA as predictors was 62.2% which, as expected, is lower than 
that (68.3%) of the analysis sample, but does not appear to be appreciably lower. Again, the 
model does improve the classification accuracy above the base rate PAC of 50% which we 
would obtain without using any predictors (if we assume the probability of being in the target 
group is 50%, as we found in the analysis sample). 

Discussion 

The results of this study support other studies which show that high school achievement 
is the best predictor of success in college (Gose, 1994; Peltier, Laden, & Matranga, 1999; Tinto, 
1993, 1998). The findings also complement those of other studies which have found a 
relationship between gender and persistence (Astin, 1993; Astin, Tsui, & Avalos, 1996; Gallicki 
& McEwen, 1989; Lewallen,1993; Daly & Breegle, 1989; and York, Dollar, & Schoob, 1993). 
The results of the current study showed that, for university freshmen who receive assistance in 
developing basic skills, gender was not a significant predictor Of retention. However, once the 
criterion included earning a first-year GPA ^ 2.5, coupled with retention, gender was a factor. 

Although SAT Quantitative score was associated with whether or not a student returned 
their second year and achieved a GPA > 2.5, it was not a significant predictor, after controlling 
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for high school GPA and gender. This finding supports the policy being considered by some 
schools of eliminating the SAT as an admission requirement, due to its lack of prediction (Cloud, 
2001; “Sad State of Testing,” 2001). The other variables available prior to admission — namely 
SAT scores— were somewhat predictive in the current study, although not in all areas. However, 
SAT was not piedictive of retention for our sample. Nor was SAT vei'bal score predictive for our 
combined outcome of GPA > 2.5 and retention after the first year. Although the SAT 
Quantitative score was associated with whether or not a student returned their second year and 
achieved a GPA > 2.5, after controlling for gender and high school GPA, the quantitative SAT 
was not a significant predictor for our population. 

Knowing which of the developmental courses students needed did not improve the 
accuracy of prediction obtained from use of pre-admission predictor variables. However, this 
population was diagnosed early (during Orientation and prior to their matriculation), then 
provided with strong intervention programs. Given that nearly half of our developmental 
education sample achieved the combined criteria (retention to second year and GPA > 2.5) 
suggests that students may enter a university under-prepared yet be successful, given appropriate 
interventions. The university provides a strong centralized developmental program, early student 
assessment, a strong faculty-referral program through its Office of Retention Services, 
supplemental instruction, a strong tutoring program, paired courses, course-based reading and 
writing stiategy training, critical thinking instruction, and a well-trained developmental 
education faculty. The developmental services are available to all students at the university, not 
just those in its developmental courses. All are practices advocated by leaders in developmental 
education (Boylan, 1999; Damashek, 1999; and Maxwell, 1997). The literature is replete with 
evidence that alternative approaches to providing developmental education, such as tutoring, 
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individualized learning laboratories, supplemental instruction, paired courses, critical thinking 
instruction, and strategic thinking instruction benefit the academic needs of the entire student 
body, and not just target those few who enter below a particular cut-off point (Boylan, Bliss, & 
Bonham, 1997; Damashek, 1999; Maxwell, 1997). 

That the type of developmental education needed was not predictive may suggest (1) all 
developmental programs (reading, writing, mathematics) at this institution are equally effective 
(or ineffective) and/or (2) that success depends more upon students’ utilization of academic 
support programs, than their specific skill deficiencies. Adelman’s (1996) work suggested that 
the type of skill deficiency does impact on students’ graduation rates; however, we find after just 
one year in college this trend is not evidenced. Further research, extending through college 
graduation for the population of developmental education students, needs to be conducted that 
incorporates the type of developmental skill(s) needed, and, the type and amount of academic 
resource utilization, in addition to pre-admission information, as predictor variables. 

The authors plan to examine the data on this group of students, after allowing for five 
years for completion of the subjects’ undergraduate studies and graduation, to see if number or 
type of developmental courses taken has any impact on their graduation rates. The examination 
of the data after graduation will, thus, more closely approximate that of Adelman (1996) and 
others. We are also interested in examining levels of participation in the university’s academic 
support services to see how well that predicts success (i.e., retention and GPA > 2.5) as well as 
graduation rates. 
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Table 1 



Descriptive statistics for continuous variables based on the full sample 
of Developmental Students (N=666) 





Minimum 


Maximum 


Mean 


Std. Deviation 


High School GPA 


2.12 


4.00 


3.22 


.41 


University GPA 


.17 


3.94 


2.56 


.66 


Verbal SAT scores 


230 


700 


463.41 


80.34 


Math SAT scores 


260 


720 


503.78 


80.87 
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Table 2 



Bivariate Correlations between Individual Predictors and Three Outcomes 



Gender 


Pearson Corr. 


University GPA 
(N=424) 

.229 


Whether Retained 
or Not (N=426) 

.097 


Whether both 
Retained and 
Univ. GPA is 2.5 
or better 
(N=426) 

.191 




Sig. (1 -tailed) 


<001 


.023 


<001 


High School GPA 


Pearson Corr. 


.408 


.213 


.360 




Sig. (1 -tailed) 


<001 


<001 


<001 


Verbal SAT scores 


Pearson Corr. 


.119 


-.089 


.069 




Sig. (1 -tailed) 


.007 


.034 


.076 


Math SAT scores 


Pearson Corr. 


.149 


.039 


.119 




Sig. (1 -tailed) 


.001 


.212 


.007 


Fail Reading Test 


Pearson Corr. 


.013 


.074 


.011 




Sig. (1 -tailed) 


.392 


.064 


.412 


Fail Writing Test 


Pearson Corr. 


-.020 


.030 


-.021 




Sig. (1 -tailed) 


.339 


.266 


.333 


Fail Math Test 


Pearson Corr. 


-.040 


-.054 


-.066 




Sig. (1 -tailed) 


.204 


.135 


.087 



Note: Bolded correlation coefficients are those significant at the 0.01 level (1 -tailed). The 
difference in sample sizes reflects two non-retained students for whom university GPA was 
unavailable. 
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Table 3 



Logistic Regression Coefficients and Related Statistics for the Model Based on the Two 
Significant Predictors of Freshman Retention with GPA ^ 2.5 





b 


SE 


Wald 


P 


Exp(b) 95.0% C.l.for Exp(h) 


Gender 


.610 


.210 


8.441 


.004 


Lower 
1.840 1.220 


Upper 

2.777 


High School GPA 


1.895 


.276 


47.283 


.000 


6.651 3.876 


1 1.415 


Constant 


-6.507 


.905 


51.733 


.000 


.001 
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Classification Accuracy Results for the Validation Sample rN=193) Using the Logistic 
Regression Model Developed on the Analysis Sample (N=445) 





Predicted Group Membership 


Percentage 

Correct 




Group 0 


Group 1 




Observed Group 0: Not Retained 

Group and/or university GPA < 2.5 


63 


31 


67.0% 


Observed 






67.0% 


Group Group 1 : Retained with 

Membership university GPA ^ 2.5 


42 


57 


57.6% 


Overall 


105 


88 


62.2% 


Percentage 









Probability 



Logistic regression curves based on two significant predictors of freshman retention 
with GPA ^ 2.5 — gender and high school GPA 
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APPENDIX 



Correspondence between tests of correlation coefficients used in the study and other commonly used inferential 
statistical procedures. 



Predictor 

Variables 


Criterion Being Predicted 


University GPA 


Retained? (l=yes) 


Retained with a Univ. 
GPA of 2.5 or better? 
(l=yes) 


Gender (l=Female) 


point-biserial 

coefficient/ 

t-test for indep. samples 


Phi coefficient / 
chi-square test of 
association 


phi coefficient / 
chi-square test of 
association 


High School GPA 


Pearson correlation 


point-biserial 

coefficient/ 

t-test for indep. samples 


point-biserial 

coefficient/ 

t-test for indep. samples 


SAT-V 


Pearson correlation 


point-biserial 

coefficient/ 

t-test for indep. samples 


point-biserial 

coefficient/ 

t-test for indep. samples 


SAT-M 


Pearson correlation 


point-biserial 

coefficient/ 

t-test for indep. samples 


point-biserial 

coefficient/ 

t-test for indep. samples 


Developmental 
Reading? (l=yes) 


point-biserial 

coefficient/ 

t-test for indep. samples 


phi coefficient / 
chi-square test of 
association 


phi coefficient / 
chi-square test of 
association 


Developmental Writing? 
(l=yes) 


point-biserial 

coefficient/ 

t-test for indep. samples 


phi coefficient / 
chi-square test of 
association 


phi coefficient / 
chi-square test of 
association 


Developmental Math? 
(l=yes) 


point-biserial 

coefficient/ 

t-test for indep. samples 


phi coefficient/ 
chi-square test of 
association 


phi coefficient / 
chi-square test of 
association 


Set of 7 Predictors 


(N/A: not our study’s 
focus) 


(N/A: not our study’s 
focus) 


logistic regression 



* ) 0 
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